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The omnipresent hands-on symbol found throughout this 
service is a constant reminder that the essence of the value 
of Microcomputers is sound analytical advice with an 
emphasis upon hands-on evaluation of business micro- 
computer software packages. This section explains how we 
provide this valuable information. Please reter to the Pack- 
ages By Vendor (800) tab for detailed charts showing the 
numeric values used in this process. 


m@ BACKGROUND 


Microcomputers have grabbed the corporate world by the 
shirt sleeves and they are taking the business professional 


with them on their upward spiral. Because of the phe- 
nomenal growth in the microcomputer marketplace, busi- 
nesses are faced with a new challenge—how to react to the 
huge influx of micro systems, micro software, and related 
micro products. Data Decisions is responding to the corpo- 
rate need tor accurate, up-to-the-minute coverage of the 
entire microcomputer spectrum with our 3-volume refer- 


ence service—MICROCOMPUTERS. 


One of the key features of this service is an expert hands-on | 


evaluation of important microcomputer software. Atter 
several months of market research, which included speak- 
ing with corporate execttives, data processing managers, 
and professional and clerical users of microcomputers, we 
repeatedly heard that the business world wants assistance 
in choosing microcomputer software. In a corporate envi- 
ronment, the cost of microcomputer software is usually not 


a problem because most packages only cost between $100 
and $500. 


However, the time involved in testing software packages to 
determine if they meet an organization's needs is very large 
and therefore very expensive. To assist corporations in their 
software evaluation process, we at Data Decisions have 
developed a unique, proprietary, precisely calibrated 
methodology to evaluate micro software in our own PC lab 
and present our results in an easy-to-interpret graph which 
can be used to compare software packages. Our method- 
ology assures you that our results are consistent, reliable, 
and in-step with the real world purpose and effectiveness 
of the software package. 


Each package is evaluated by our software analysts in our 
PC lab. The lab is currently equipped with several IBM 
PC/XTs, an Apple II and II], a DEC Rainbow, and several 
IBM PC-compatible systems. In addition, we maintain a 
library of popular operating systems in several versions 
and a wide range of low-, medium-, and high-performance 
peripheral devices. This gives us the opportunity to con- 
figure a system to run virtually all of the available micro- 
computer software and to take advantage of the features of 
these packages. As new systems, software, and peripherals 
become available, we add these to our laboratory. 
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Our analysts are specialists in several categories of micro- 
computer software. In order to insure that our evaluations 
are valid and consistent among reports, we have developed 
a certification and training program. Before an analyst 
writes a software evaluation for publication, a sample report 
must be submitted to an editorial review board for approval. 
This review board consists of 2 to 3 corporate users of a 
particular software package. One specific package has 
been chosen in each of the major categories of software; 
this package serves as a ‘signature piece” to qualify our 
analysts. The categories are: 

® word processors 

® spreadsheet programs 

® database programs 

® graphics programs 

® communications programs 


Upon receipt of approval of the evaluation, the analyst is 
then certified to review other software packages in the 
same category of software for which approval was granted, 
for example, word processors, graphics, spreadsheets, etc. 
This procedure must be repeated for each additional cate- 
gory of software for which certification is needed. Our cer- 
tification program guarantees that the analyst thoroughly 
understands the functions of each package, and more 
importantly that the analyst can bring a consistent, appro- 
priate corporate type of perspective to each evaluation. 


The evaluation process combines the talents of our profes- 
sional, technical, and clerical staff. The professional staff 
consists of those editors and senior editors who have 
received certification from the editorial review board. The 
technical staff includes programmers and editorial per- 
sonnel with a background in software development, system 
design, computer science, or programming. The clerical 
staff consists of secretaries, our research staff, and assistant 
editors. : | : | 


All packages are rated on 7 characteristics: Environment, 
Documentation, Functionality, Ease of Use, Support, System 
Interface, and Experience of Vendor. Each of 5 categories 
of sottware—Word Processors, Spreadsheets, Data Man- 
agement, Graphics, and Communication—is rated on dif- 
ferent criteria explicitly related to the package's particular 
function. Please refer to the report behind the Packages By 
Vendor (800) tab for an explanation of each of our rating 
criteria and the quantitative methodology used to score the 
packages. _ 


The first step in each evaluation is always performed by a 
member of our technical staff. The staff member loads the 
program diskettes and rates the particular package in the 
areas of Environment and System Interface. The technical 
staff member may also make appropriate comments on the 
Ease of Use criteria. When the system is up and running, 
the remaining quantitative scoring for the Ease of Use, 
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Documentation, and Functionality sections is usally per- 
formed through the combined efforts of our clerical, tech- 
nical, and professional reviewers, depending on the par- 
ticular category of software. The professional staff member 
is always available to the lab during this evaluation. 


At the conclusion of the quantitative rating of the package, 
which is usually performed by the clerical and technical 
staff, the senior person carefully reviews the scores and 
will ring the software package up and verify many of the 
scores; particularly in the areas of Ease of Use and Func- 
tionality. This analyst then prepares the written report in 
consultation with the other staff members involved in the 
evaluation. He draws on knowledge of the micro software 
market, software trends, and knowledge of other software 
packages to create the Analysis, Strengths, Limitations, 
Hands-On Evaluation, and Experience of Vendor sections 
of the report. 


@ DETAILED EVALUATION PROCEDURES 


The functions provided by each different type of software 
package obviously vary. Therefore, the criteria by package 
type we use to evaluate each type is, of necessity, different. 
In the paragraphs that follow are included brief descrip- 
tions of the methodology used to evaluate the software, 
together with some short scenarios employed as aids in the 
evaluation process. 


For each of the 5 major categories of software, we have 
included a discussion of important characteristics evalu- 
ated for each section to arrive at the Product Quality Rat- 
ings. The method for evaluating 2 sections in the Product 
Quality Ratings—Support and Experience of Vendor—did 
not vary from software package category to category; and 
consequently, is not included in the paragraphs below. 


To evaluate Support for each package we looked for the 


presence of a vendor-supplied Technical Support Group; . 


we determined if a telephone hot line was available or an 
800-telephone number, we determined how much the 
vendor relied on the distributor for support; and we looked 
for a software update policy. 


The major features in the Experience of Vendor category 
were the number of years the vendor was in the micro 
marketplace; the number of micro software products offered; 
if the vendor had experience in developing and marketing 
mainframe software; and the quality of management. 


For a more detailed description of our methodology and a 
listing of our numeric point structure used to arrive at the 
Product Quality Ratings please refer to the Software Evalu- 
ation report behind the Packages By Vendor (800) tab. 


[] Word Processor Packages 


Environment @ We iried to pay particular attention to the 
disk and memory requirements and their effect on perfor- 
mance. We noted if the package ran on the configuration 
typically sold and if it used the most current operating 
system version. We also tested operation of the package on 
minimum memory by searching a file of at least 1OK bytes 
to see if the time to find a word at the end of a document 
was excessive with small memory systems. 

Documentation ¢ Most word processors need a good refer- 
ence manual which is organized by user functions and not 
by the command to invoke them. Other helpful aids are a 
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reference card or guide, a pocket reference, and a tutorial 


manual/diskette. A package received a high score for 
documentation if all three types were available and were 
well-organized for their specific function. 


Functionality @ We looked for specific features in all word 
processing packages including: full-screen editing, a for- 


mat of the display matching the printed page, on-screen 


page and line indications, and global search and replace, 
among others. For a full listing of scored features please 
refer to the report behind the Packages By Vendor (800) 
tab. In order to test all functions our clerical staff keyed in 
several documents prepared by our professional staff. 
These documents were then revised based on ‘'standard- 
ized revisions.’ These ‘standardized revision” documents 
were also prepared by our professional staff to insure that 
all functions of word processing packages were tested. 


Ease of Use @ The key points that we looked for in this area 


were the richness of the “Help” function and the quality of 
the command structure. A hierarchical “Help” structure 


with aid at each point is desirable; we rated a package 
lower if the “Help” structure required the user to know the 
command name to request help on it. In command struc- 
ture, menu lists should be used for non-editing inputs such 
as the selection of files to be edited or basic pre-edit options. 


System Interface e Word processors must frequently operate 
in an environment with other types of word processors or 
with computer systems which store text. We awarded points 
for the presence of detailed formats of the document struc- 
ture, for instructions on converting this format to that of 
other popular systems, and for the ability to read page- 
formatted documents and convert them to word process 
format. 


O Spreadsheet Software 


Environment ® Our scoring criteria was based on the fact 
that we believe spreadsheet programs should operate ona 
system with a single disk drive and should take advantage 
of additional memory to provide larger in-core storage of 
sheets. We awarded points if a system required only one 
disk drive for operation and operated in less than 64K 
bytes of memory. We scored additional points if the pro- 
gram could use 128K bytes or more of RAM. 


Documentation @ We consider this an important character- 
istic for spreadsheet programs since there are usually many 
functions that require description. We believed there 
should either be two manuals for the spreadsheet packages 
or a tutorial section in the reference manual. It was also 
important that the reference documentation be organized 
in functional lines rather than by command, and that there 
be a list of commands and an indication of their use in a 
special section. 


Functionality ¢ We determined that spreadsheets should 
have the following functions among others: the ability to 
use cursor control to move left/right and up/down from | 
cell to cell: the ability to go to a selected cell, to the upper 
lett or lower right of the worksheet; or to the left or right end 
of the current row or top or bottom of the current column. 
We also feel that spreadsheets should permit the entry of 
numeric or alphabetic data into a cell and should permit 
the naming of a particular cell and the use of the name as a 
reference to the cell. The report behind the Packages By 
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Vendor (800) tab will indicate all other features which we 
evaluated. 


In order to give this section of the evaluation a corporate 
oerspective our professional staff developed real-world 
business applications to test all of the functions. Our clerical 
staff entered the data and noted on a worksheet the pres- 
ence or absence of particular features. When value judg- 
ments were required, the professional staff member was 
consulted. 


Ease of Use ® Since spreadsheets are rich in commands 
and often difficult to master the Help function is very 
important and should be usable to provide both a list of 
commands and the detail on any command. Data Decisions 
believes that the commands should be in menu form rather 
than tree-form entry since there are many types of com- 
mands. We also feel that the display form should resemble 
a normal! spreadsheet with the addition of row and column 
references. Points were awarded in the 3 areas of “Help” 
function, command structure, and display format. 


System Interface @ Because of the complex structure of 
spreadsheets, there is little compatibility in interface be- 
tween spreadsheets and other systems. However, we 
believed there should be an explanation of the file format 
for spreadsheet files provided with the package, or the 
package should provide the option of producing files 
compatible with the normal files of the computer on which 
it is run. 


CL] Data Management Packages 


Environment @ We determined that a data management 
package normally requires 2 disks and 64K bytes of 
memory for operation. If the system operates with a single 
drive, additional points were awarded. Extra points were 
given if the system supported 128K bytes or more of 
memory, 


Documentation @ Our analysts felt that 2 manuals were a 
necessity. A tutorial manual with an accompanying dem- 
onstration program diskette should be provided to learn 
the system, and a reference manual with detailed com- 
mand information should be available when the user is 
experienced. We believed the documentation should be 
organized with the realization that the user is necessarily 
more technical than the average user of a word processor. 
The tutorial manual/diskette should provide some intro- 
ductory information on the concepts of the data manage- 
ment program and then move to build user confidence 
and experience through the use of successively more 
complex examples. The reference manual should provide 
a grouping of commands by their functions, but a reference 
of commands by command code word is also available. 


Functionality © Once the package was loaded by our tech- 
nical staff, our clerical and professional analysts worked 
closely together. We examined 3 areas of data manage- 
ment functions: file maintenance, report generation, and 
special programming. In the file maintenance area we 
looked for the ability to design data entry screens, to define 
input field data types with editing for proper data, to define 
a record with a large number of fields, and for field size 
limits of 255 or more characters. In the report generation 
area we looked for automatic totaling, control break 
recognition with subtotals, column heading specification 
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including multiline headings, and specification of column 
width. In the special programming support area we tested 
for the ability to test field values (the IF command), and to 
maintain internal data areas or indicators. For a complete 
list of features and scoring criteria for this section please 
refer to the pages under Data Management in the Software 
Evaluation section behind the Packages By Vendor (800) 
tab. 


Ease of Use @ In this section, we determined that the most 
important feature was the structure of the operator com- 
mands. We believe that commands should be in a roughly 
English syntax and should extend to multiple lines if 
required. Error messages should indicate the area where 
syntax or usage is suspect and not just reject the command. 
Other important features are the ability to define “com- 
mand files” where lists of commands are stored for invoca- 
tion as a single stream, and the ability to enter and update 
screen and report formats in a convenient way. In order to 
score this area our professional and clerical stati worked 
closely. In some cases our professional staff member had to 
translate error messages; in every case it was the profes- 
sional member's responsibility to define and test the 
“command files” where lists of commands were stored for 
invocation as a single stream. 


System Interface @ Data management files may be derived 
from input entered elsewhere or produced on the system 
but by other programs. We believe that the ability to 
manipulate database files was highly desirable, as was the 
ability to gather data from other popular systems on main- 
frame computers. We awarded points if the database file 
structures were defined for reference by programs written 
in other languages and if the file structure was designed to 
facilitate such use. We also gave points if there was a facility 
to produce files in non-database format out of the database 
package, and if the user could read such files and generate 
database files. We also awarded points if the package was 
integrated with at least one mainframe program for file 
exchange and if specific guidelines for cross-exchange of 
files was supported. 


[1] Graphics Software 


The elevation of business graphics software packages 
presented an interesting challenge because of the wide 
range of facilities available. We decided to divide the 
graphics program into 2 main categories: programs to 
produce graphs from spreadsheet or database information, 
and programs to generate graphic images “free-form” for 
presentations or reports. 


Environment @ Our analysts consider that the major envi- 
ronmental question for graphic programs is the type of 
graphics equipment which is required for it to run. It is 
desirable that product-line hardware be used wherever 
possible rather than custom hardware, and that any 
equipment not provided by the computer manufacturer 
should come from a reputable and reliable source. 


Documentation @ We believe that software for graphing of 
spreadsheet data should be easily operated and have little 
need for extensive documentation. The manual should out- 
line the graphing options and indicate the way in which 
each is selected. It should give image-oriented examples 
of the appearance of each type of graph available and the 
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effect of each variable option which the user can control. It 
should also provide suggestions on selecting options to 
match the characteristics of the data or the way in which 
the presentation is to be used. 


Graphics software used for es: form creation of images 
should have extensive tutorial and reference documenta- 


tion which teaches the basic use of the package and. 


defines the detailed operation of commands. The tutorial 


material should ideally be accompanied by diskette media. . 


Illustrations are vital in graphics manuals, and they should 
be large and accurately reflect the screen appearance. 
Use of external devices such as plotters and printers should 
also be covered. 


Functionality @ Our professional staff developed one 
“standard” graph that was to be created from each 
spreadsheet and database program, and another ''stan- 
dard” graph which we used as a basis for the “free-form” 
graphics packages. It was against these standard” graphs 
that we tested the features of each particular package. For 
example, on graphs developed trom spreadsheets our pro- 
fessional and clerical staff developed a number of graph 
types—line, pie, and bar charts; color was changed; and 
the horizontal and vertical scales were changed, etc. For 
image graphics packages we considered that important 
features were: the ability to draw free-form images, the abil- 
ity to generate geometric shapes at various points and at 
specified sizes, the ability to change colors and to fill in 
spaces with a particular color. The total number of colors in 
the palette was also considered important. For a complete 
list of features evaluated please refer to the Software Evalu- 
ation report behind the Packages By Vendor (800) tab. 


Ease of Use @ Our analysts believe that spreadsheet graph- 
ics packages are easy to use if they can be easily invoked 
and if the graph options are pre-selected where possible 
and menu or prompt chosen otherwise. Those which-are 
directly invoked from the spreadsheet are easiest to use. 
We believe that image packages are more easily used if 
they accept a form of analog drawing information from a 
mouse, joystick, or graphics tablet. They should also have a 
menu of functions and a Help function with commands. 


System Interface @ For this we award points on spreadsheet 


graphics if the package graphed normal spreadsheets: if it 
works with multiple spreadsheet programs, and if it would 
work with other types of programs. For image graphics we 
award points for support of each standard graphics protocol 
supported such as Tektronix, HP, IBM color 3270, and 
NAPLPS videotex, and additional points for the quality of 
instructions on the use of the graphics data with other non- 
supported hardware. 


Communications Software 


Our technical staff played an integral role in evaluating 
the Communications packages. In fact, a technical staff 
member with communication experience usually took 
responsibility for a large part of the scoring section of this 
evaluation. With these packages we found remote sites with 


which to communicate by setting up communication links 


with several host computers, usually DEC or IBM systems. 
We evaluated each package based on the type of facility 
which it provides: terminal emulation, transparent file 
transfer, or program file transfer. 
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Environment @ The ess questions in this category are the 
ability of the package to support product line hardware’ 
and. the typical configuration of the system. We award 
points if the system runs on the normal configuration of the 
target computer; if it runs with the normal operating system; 
if it effectively utilizes additional memory to improve per- 
formance; and if all the hardware required is supplied by 
the computer vendor. 


Documentation @ We feel that 3 areas of documentation are 
important: the communication link, the local operation, and 
the remote system operation. For the communication oper- 
ation, we awarded points if the document defines commu- 
nication terms and explains the various communication 
options, modems, etc which are associated with the pack- 
age, if a detailed explanation of setup of the communication 
line is included and if the errors and problems which might 
be encountered are outlined and explained. In the local 
operation section, we award points if the manual/diskette 
explains not only how to load and run the program but also 
how to perform the operation of the device which was emu- 
lated on the PC, if a description of loading and running the. 
program is present and if there is a mapping of “target 
terminal” keys to the PC keyboard. For remote system 
operation, points are given if the documentation defines 
the set up required at the remote system, if the error condi- 
tions of the remote system are related to the local package 
operation, and if tips are available on using the remote 
system to diagnose problems on the connection. 


Functionality @ Our analysts determined that because 
communication packages could have many functions, it 
would be difficult to rate functionality. Therefore, we 
determined which of the 3 functional areas—terminal 
emulation, transparent file transfer, or programmed file 
transfer—the package was intended to support and then 
rated the package in those areas. Please see the description 
of features in the Software Evaluation section behind the 


Packages By. Vendor (800) tab. 


Ease of Use @ We rated three areas: the ease in setting up 
the connection to the other system using the package, 
including the special procedures on the other system; the 
ease with which an operator trained in the use of the target 
terminal would adjust to the new package; and the ease 
with which non-emulator functions such as file transfers 
could be invoked: We also give an extra point if the 
package allows the user to escape from the communica- 
tion circuit to gain access to function menus or help 
documentation. | | 


System Interface @ Our analysts aan that the rating 
should reflect the universality of the communication inter-_ 
face. We therefore award points for asynchronous, bi- 
synchronous 3270 interface, SNA 3270 interface, or 
2780/3780 support; for support of multiple codes sets; for 
support of transmission speeds from 1200 to 9600 bps; and 
if the product can share a line with other devices. | 


O Multifunction Packages 


Multifunction packages present a unique challenge. We. 
decided that our professional staff members should evalu- 
ate the whole package themselves with no clerical help. 
This decision was made because it was determined that 


some multifunction packages are in truth merely spread- 
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sheet packages with some data management and graphics 
facilities; while other packages have completely integrated 
several functions. (We believe the micro user will increas- 
ingly see more of the latter type of software in the near 
future.) 


Therefore, in order to evaluate accurately each multifunc- 
tion package, a professional staff member has complete 
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responsibility. lf a multifunction package is determined to 
provide primarily spreadsheet functions, it will be evalu- 
ated using the spreadsheet criteria. If it provides primarily 
data management functions, those criteria will be used, 
etc. If a package is truly integrated, the characteristics for 
each function will be evaluated and reported separately. 


e END 
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